Three-Dimensional Computer Graphics Rendering Algorithm for Simulation of Vision from Sparse Shack-Hartmann Sensor Output Data

The goal of this project is to develop methods to demonstrate how the three-dimensional world appears to people who have different types of aberrations. The input information is the subject's wavefront aberration measurements and a scene where the locations and depth of all objects are known. The first step of our approach is to replace the continuous depth map with a discrete map sampled at approximately 1/4 diopter steps, which is close to the human blur resolution threshold. A depth point spread function (DPSF) for each depth plane is calculated based on the subject’s wavefront aberration measurements. The DPSF for a given depth plane is then convolved with the portion of the image that corresponds to that depth plane, with appropriate compensation for occlusion edges, in order to produce a blurred version of that portion of the image. The multiplicity of these blurred images fragments are then composited to form the blurred version of the complete image. The algorithm is much faster than ray tracing algorithms of comparable image quality. 1. Overview Our recent work which we call vision-realistic rendering [Barsky2001, Barsky2002] develops new three-dimensional rendering techniques for the computer generation of synthetic images and animations that will simulate the vision of a subject whose wavefront aberrometry we have measured. We are implementing vision-realistic rendering in a prototype system called RAYS (Render As You See), which can simulate the vision of a particular individual. Each input image can be anything – a photograph, a computer-generated 2D image, a frame from an animation, or even a standard Snellen visual acuity eye chart – as long as there is accompanying depth information. Our approach takes three types of information as input: (1) a color input image where the entire 3D scene is in focus and there is no blur (either computer-generated or taken with a pinhole camera), (2) a depth map specifying the depth of each point of the input image, (3) the observer's wavefront aberrations. The wavefront specifies the phase of the light in the plane of the entrance pupil coming from a single retinal point. Our goal is to use these three pieces of information to appropriately blur the input image (item 1) taking into account the dioptric blur corresponding to the object distance and the observer's aberrations. One of the purposes of this project is to demonstrate how the world appears to people who have different different types of aberrations. Recently, there has been an interest in demonstrating how corneal refractive surgery can reduce the eye's aberrations. A possible disadvantage of reducing aberrations is that it is possible for the aberrations to improve depth of field similar to the action of a bifocal contact lens. Our methods allow us to visualize how the aberrations and depth of field interact when viewing realistic scenes We begin by extracting wavefront aberration measurements from the output of a Shack-Hartmann wavefront aberrometry device [Platt1971] (as shown in Figure 1) that measures the aberrations that are present in the optical system of the subject. Our approach is to blur sharp images based on depth information along with the optical aberrations of a particular individual, as well as a specified focusing distance. From the wavefront, we would ideally determine a continuous blur function, but instead we discretize the function by sampling at a set of focusing distances. We specify these distances as a set of depth planes that discretize the three-dimensional space. For each depth plane, we construct a blur filter in object space from sparse Shack-Hartmann sensor output data. Figure 1. The first author’s eye being measured by the Shack-Hartmann wavefront aberrometry device. 2. Ray Optics vs. Wave Optics The present RAYS software constructs the OSPSF using ray optics. A more accurate OPSF would use wave optics and would include chromatic aberration and Stiles-Crawford apodization [Atchison 2002]. However, our goal in the present paper was not to calculate the optimal OSPSF, but rather to demonstrate how one can use a bank of multiple-depth DPSFs to render three-dimensional scenes where there are large variations in depth. Calculating wave-DPSFs is more time consuming than calculating ray-DPSFs since in the former case this has to be done using multiple wavelengths to smooth out the rapid diffractive oscillations that are produced by the sharp edge of the pupil. The wave-optics method has the advantage that the OSPSF can be calculated using the rapid fast-Fourier transform (FFT) method. disadvantage of the FFT method is that in the presence of aberrations, the wave phase can change rapidly and consequently very fine sampling across the pupil is needed. In the future, we intend to compare the ray-OSPSF and wave-OSPSF in the presence of aberrations and blur to determine the conditions where the two differ and to determine the practical constraints on the two methods. 3. Outline of the Algorithm (a) (b) Figure 2. (a) Rays are cast from a point light source on the retina and pass through a virtual lens, thereby creating the measured wavefront. This wavefront is sampled and rays are cast normal to it. The DPSFs are determined by intersecting these rays at a sequence of depths (b) Construction of the set of DPSFs from a dense set of normal vectors sampled from the wavefront. The ray leaving the pupil going downward to the left, has a positive slope in our coordinate system. Figure 3. Wavefront and corresponding blur filter. 3. Example Images Figures 4-8 are vision-realistic renderings of a room scene. Figure 4 is a simulation that models ideal vision and Figures 5-7 are simulations of the vision of actual individuals based on their measured data. Notice that the nature of the blur is different in each image. The field of view of the image is approximately 46o and the pupil size is rather large at 5.7 mm. For Figure 4, we constructed an OSPSF from a planar wavefront to yield a simulation of vision for an aberration-free model eye. The simulation of vision shown in Figure 5 is based on the data from the left eye of male patient GG who has astigmatism. Note how the blur is most pronounced in one direction (in this case horizontal), which is symptomatic of astigmatism. Figure 4: Simulation of vision of aberration-free model eye. Figure 5: Simulation of vision of astigmatic patient GG. Next, we show vision-realistic rendered images based on pre-and post-operative data of patients who have undergone LASIK vision correction surgery. Specifically, the vision for the right eye of male patient DB is simulated in Figure 6, and then Figure 7 simulates the vision of the left eye of male patient DR. For each patient, the pre-operative vision is simulated in the top image while the lower image simulates the post-operative vision. The images demonstrating pre-operative vision show the characteristic extreme blur pattern of the highly myopic (near-sighted) patients who tend to be the prime candidates for this surgery. Although, in both cases, the vision has been improved by the surgery, it is still not as good as the aberration-free model eye. Furthermore, the simulated result of the surgery for patient DB is slightly inferior to that depicted for patient DR. However, note that the patient (DB) with the inferior surgical result had significantly inferior pre-operative vision com-pared to that of patient DR. Figure 6: Simulation of vision of LASIK patient DB Figure 7: Simulation of vision of LASIK patient DR based on (a) Pre-operative and (b) Post-operative data. based on (a) Pre-operative and (b) Post-operative data. Figure 8 is computed based on data measured from the left eye of female patient KS who has the eye condition known as keratoconus. This image shows the distortion of objects that is caused by the complex, irregular shape of the keratoconic cornea. Note how the nature of these visual artifacts is distinct from what would generally be a smooth blur that occurs in more pedestrian vision problems such as myopia (see Figure 7(a)). This distinction is often not understood by clinicians when it is articulated by keratoconic patients. We hope our techniques could be used in optometry and ophthalmology for the education of students and residents as well as for the continuing education of clinicians. Figure 8: Vision-realistic rendered image simulating vision based on actual wavefront data from a patient with keratoconus. Finally, Figure 9 shows several frames from a rack focus applied to a shot from Pixar’s short film Tin Toy. The original short film was rendered without depth of field. We add depth of field to this shot to draw the audience’s attention from the Tin Toy to the baby as he enters the scene. The depth information for this film no longer exists; thus, we hand-generated the depth map. We deduced that the field of view was approximately 21 1/3o and from that we determined that the focal length of the lens would be about a 100 mm telephoto lens (assuming a 35 mm film format). In these frames, we set the aperture to f/2.8. Figure 9: Frames from a rack focus sequence using a shot from Pixar’s Tin Toy as input. To consider the computational requirements of our technique, note that it comprises three parts: fitting the wavefront surface, construction of the OSPSF and the rendering step. The computation time for the surface fitting is negligible. The time to compute the OSPSF depends on the number of wavefront samples. For the images in this paper, the computation of the OSPSF, using one million samples, was less than half a minute. The rendering step is dominated by the FFTs performed for convolution (our kernels are generally too large to convolve in the spatial domain). Thus, the computation time for the rendering step is dependent on the image size and the number of non-empty depth images. The room scene has a resolution of 1280 X 720 and each image took about 8 minutes to render, using 11 depth images, on a Pentium 4 running at 2.4 GHz. AcknowledgmentsThe authors are indebted to Adam Bargteil, Dan Garcia, Ph.D., Viet Dinh, and Gabriel N. Garza in the Computer Science Division of theUniversity of California, Berkeley for their numerous contributions to this work, and are grateful to Ian Cox, Ph.D., and Michele Lagana,O.D., at Bausch and Lomb for providing the Shack-Hartmann data used in this paper.This work was supported in part by the National Science Foundation award number ASC-9720252, “Visualization and Simulation inScientific Computing for the Cornea,” award number CDA-9726362, “Virtual Environments for Telesurgery and Surgical, Training:Efficient Computation, Visualization, and Interaction”, and award number CCR-0209574, “Vision-Realistic Rendering”. References[Atchison2002] David A. Atchison, Dion H. Scott, Niall C. Strang, and Pablo Artal (2002) Influence of Stiles-Crawford apodization onvisual acuity. J. Opt. Soc. Am. A, 19(6): 1073-1083, 2002. [Barsky 2001] Barsky, Brian A.; Garcia, Daniel D.; and Klein, Stanley A. (2001) "Computer Simulation of Vision-Based Synthetic ImagesUsing Hartmann-Shack-Derived Wavefront Aberrations", Association for Research in Vision and Ophthalmology, Fort Lauderdale, Florida,29 April 4 May 2001. Abstract in Investigative Ophthalmology & Visual Science, Vol. 42, No. 4, March 15, 2001, pp. S162. [Barsky2002] Brian A. Barsky, Adam W. Bargteil, Daniel D. Garcia, and Stanley A. Klein, Introducing Vision-Realistic Rendering,Thirteenth Eurographics Workshop on Rendering, poster paper proceedings, pages 1-7, Pisa, June 26-28, 2002. [Platt1971] Ben C. Platt and Roland V. Shack. Lenticular Hartmann-screen. Newsletter 5, 15, Optical Science Center, University ofArizona, 1971.